-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add deferrable mode to MLEngineStartTrainingJobOperator #27405
Conversation
894932b
to
9c0ddf2
Compare
conflicts :( |
46de5e1
to
c690fba
Compare
c690fba
to
f9a2951
Compare
962922e
to
22f97d3
Compare
22f97d3
to
61acbaa
Compare
Hello @potiuk |
For : https://github.com/apache/airflow/actions/runs/3523584902/jobs/5908291978 - the steps to follow are described in the instructions after the error is displayed. Just follow. The problem is really about new "warning" generated and you just need to add the warning as "known" warning to the list of known warnings. The other failures are - I think - just intermitteent errors - this seems to be a problem when you run the whole suite of tests as a non-commiter, sometimes (and we have not figured out when) the tests are failing intermittently in groups - but when we re-run them they work. those are flaky tests that we need to fix eventually. Please rebase and fix the warning and if other tests fail next time, ping me and I will re-run them. |
f86e84b
to
103f21a
Compare
@potiuk |
Yeah. I will take a look at those failing "public runners" tests shortly (cc: @Taragolis - unless you want to take a look following my description). |
Restarted. |
9f66c08
to
115a89e
Compare
115a89e
to
6b2291d
Compare
Rebased after fixing the "public" tests last week. |
@potiuk |
Interesting, the reason of failing tests could be some kind of race condition. |
Seems like the issue come from this class, which changed by this PR: #28047: airflow/airflow/executors/kubernetes_executor.py Lines 66 to 77 in 672264b
During the tests time to time additional namespace |
@Taragolis |
It started to happen recently randomly in a number of PRs and usually goes away afer re-running, typical flakey test |
@XD-DENG - maybe you can also take a look to see for a possible race condition it could have caused? It seems like a side-effect - especially that when it fails it fails because ir returns several namespaces (airflow and default) so it is rather appropriate for the multi-namespace change being the culprit. |
Sure I will take a look @potiuk . Please ping me for reminding if you don't hear from me later |
OK. I fixed it (and I found that the test missed one assert). PR is coming. |
Fix to the flaky exception here: #28475 |
@potiuk |
6b2291d
to
a25059c
Compare
@potiuk |
@potiuk |
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.